Light sensitive display

ABSTRACT

A light sensitive display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.11/901,649, filed Sep. 18, 2007, which is a continuation of U.S. patentapplication Ser. No. 10/371,413, filed Feb. 20, 2003, now U.S. Pat. No.7,280,102, which is a continuation of U.S. patent application Ser. No.10/442,433, filed May 20, 2003, which claims the benefit of U.S.Provisional App. No. 60/383,040, filed May 23, 2002 and U.S. ProvisionalApp. No. 60/359,263, filed Feb. 20, 2002.

BACKGROUND OF THE INVENTION

The present invention relates to touch sensitive displays.

Touch sensitive screens (“touch screens”) are devices that typicallymount over a display such as a cathode ray tube. With a touch screen, auser can select from options displayed on the display's viewing surfaceby touching the surface adjacent to the desired option, or, in somedesigns, touching the option directly. Common techniques employed inthese devices for detecting the location of a touch include mechanicalbuttons, crossed beams of infrared light, acoustic surface waves,capacitance sensing, and resistive materials.

For example, Kasday, U.S. Pat. No. 4,484,179 discloses anoptically-based touch screen comprising a flexible clear membranesupported above a glass screen whose edges are fitted with photodiodes.When the membrane is flexed into contact with the screen by a touch,light which previously would have passed through the membrane and glassscreen is trapped between the screen surfaces by total internalreflection. This trapped light travels to the edge of the glass screenwhere it is detected by the photodiodes which produce a correspondingoutput signal. The touch position is determined by coordinating theposition of the CRT raster beam with the timing of the output signalsfrom the several photodiodes. The optically-based touch screen increasesthe expense of the display, and increases the complexity of the display.

Denlinger, U.S. Pat. No. 4,782,328 on the other hand, relies onreflection of ambient light from the actual touch source, such as afinger or pointer, into a pair of photosensors mounted at corners of thetouch screen. By measuring the intensity of the reflected light receivedby each photosensor, a computer calculates the location of the touchsource with reference to the screen. The inclusion of the photosensorsand associated computer increases the expense of the display, andincreases the complexity of the display.

May, U.S. Pat. No. 5,105,186, discloses a liquid crystal touch screenthat includes an upper glass sheet and a lower glass sheet separated byspacers. Sandwiched between the glass sheets is a thin layer of liquidcrystal material. The inner surface of each piece of glass is coatedwith a transparent, conductive layer of metal oxide. Affixed to theouter surface of the upper glass sheet is an upper polarizer whichcomprises the display's viewing surface. Affixed to the outer surface ofglass sheet is a lower polarizer. Forming the back surface of the liquidcrystal display is a transflector adjacent to the lower polarizer. Atransflector transmits some of the light striking its surface andreflects some light. Adjacent to transflector is a light detecting arrayof light dependent resistors whose resistance varies with the intensityof light detected. The resistance increases as the light intensitydecreases, such as occurs when a shadow is cast on the viewing surface.The light detecting array detect a change in the light transmittedthrough the transflector caused by a touching of viewing surface.Similar to touch sensitive structures affixed to the front of the liquidcrystal stack, the light sensitive material affixed to the rear of theliquid crystal stack similarly pose potential problems limiting contrastof the display, increasing the expense of the display, and increasingthe complexity of the display.

Touch screens that have a transparent surface which mounts between theuser and the display's viewing surface have several drawbacks. Forexample, the transparent surface, and other layers between the liquidcrystal material and the transparent surface may result in multiplereflections which decreases the display's contrast and produces glare.Moreover, adding an additional touch panel to the display increases themanufacturing expense of the display and increases the complexity of thedisplay. Also, the incorporation of the touch screen reduces the overallmanufacturing yield of the display.

Accordingly, what is desired is a touch screen that does notsignificantly decrease the contrast ratio, does not significantlyincrease the glare, does not significantly increase the expense of thedisplay, and does not significantly increase the complexity of thedisplay.

The foregoing and other objectives, features, and advantages of theinvention will be more readily understood upon consideration of thefollowing detailed description of the invention, taken in conjunctionwith the accompanying drawings.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a cross sectional view of a traditional active matrix liquidcrystal display.

FIG. 2 is a schematic of the thin film transistor array.

FIG. 3 is a layout of the thin film transistor array of FIG. 2.

FIGS. 4A-4H is a set of steps suitable for constructing pixel electrodesand amorphous silicon thin-film transistors.

FIG. 5 illustrates pixel electrodes, color filters, and a black matrix.

FIG. 6 illustrates a schematic of the active matrix elements, pixelelectrode, photo TFT, readout TFT, and a black matrix.

FIG. 7 illustrates a pixel electrode, photo TFT, readout TFT, and ablack matrix.

FIG. 8 is a layout of the thin film transistor array of FIGS. 6 and 7.

FIG. 9 is a graph of the capacitive charge on the light sensitiveelements as a result of touching the display at high ambient lightingconditions.

FIG. 10 is a graph of the capacitive charge on the light sensitiveelements as a result of touching the display at low ambient lightingconditions.

FIG. 11 is a graph of the photo-currents in an amorphous silicon TFTarray.

FIG. 12 is a graph of the capacitive charge on the light sensitiveelements as a result of touching the display and providing light from alight pen.

FIG. 13 is an alternative layout of the pixel electrodes.

FIG. 14 illustrates a timing set for the layout of FIG. 13.

FIG. 15 illustrates a handheld device together with an optical wand.

FIG. 16 illustrates even/odd frame addressing.

FIG. 17 illustrates a front illuminated display.

FIG. 18 illustrates total internal reflections.

FIG. 19 illustrates a small amount of diffraction of the propagatinglight.

FIG. 20 illustrates significant diffraction as a result of a plasticpen.

FIG. 21 illustrates a shadow of a pointing device and a shadow withilluminated region of a pointing device.

FIG. 22 illustrates a modified black matrix arrangement.

FIG. 23 illustrates a light reflecting structure.

FIG. 24 illustrates a pen.

FIG. 25 illustrates a light guide and finger.

FIG. 26 illustrates a display with multiple sensor densities and opticalelements.

FIG. 27 illustrates a display with memory maintaining material.

FIG. 28 illustrates another pen.

FIG. 29 illustrates another pen.

FIG. 30 illustrates another pen.

FIG. 31 illustrates another pen.

FIG. 32 illustrates another light guide.

FIG. 33 illustrates an image acquisition and processing technique.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENT

Referring to FIG. 1, a liquid crystal display (LCD) 50 (indicated by abracket) comprises generally, a backlight 52 and a light valve 54(indicated by a bracket). Since liquid crystals do not emit light, mostLCD panels are backlit with fluorescent tubes or arrays oflight-emitting diodes (LEDs) that are built into the sides or back ofthe panel. To disperse the light and obtain a more uniform intensityover the surface of the display, light from the backlight 52 typicallypasses through a diffuser 56 before impinging on the light valve 54.

The transmittance of light from the backlight 52 to the eye of a viewer58, observing an image displayed on the front of the panel, iscontrolled by the light valve 54. The light valve 54 normally includes apair of polarizers 60 and 62 separated by a layer of liquid crystals 64contained in a cell gap between glass or plastic plates, and thepolarizers. Light from the backlight 52 impinging on the first polarizer62 comprises electromagnetic waves vibrating in a plurality of planes.Only that portion of the light vibrating in the plane of the opticalaxis of the polarizer passes through the polarizer. In an LCD lightvalve, the optical axes of the first 62 and second 60 polarizer aretypically arranged at an angle so that light passing through the firstpolarizer would normally be blocked from passing through the secondpolarizer in the series. However, the orientation of the translucentcrystals in the layer of liquid crystals 64 can be locally controlled toeither “twist” the vibratory plane of the light into alignment with theoptical axes of the polarizer, permitting light to pass through thelight valve creating a bright picture element or pixel, or out ofalignment with the optical axis of one of the polarizes, attenuating thelight and creating a darker area of the screen or pixel.

The surfaces of the a first glass substrate 61 and a second glasssubstrate 63 form the walls of the cell gap are buffed to producemicroscopic grooves to physically align the molecules of liquid crystal64 immediately adjacent to the walls. Molecular forces cause adjacentliquid crystal molecules to attempt to align with their neighbors withthe result that the orientation of the molecules in the column ofmolecules spanning the cell gap twist over the length of the column.Likewise, the plane of vibration of light transiting the column ofmolecules will be “twisted” from the optical axis of the first polarizer62 to a plane determined by the orientation of the liquid crystals atthe opposite wall of the cell gap. If the wall of the cell gap is buffedto align adjacent crystals with the optical axis of the secondpolarizer, light from the backlight 52 can pass through the series ofpolarizers 60 and 62 to produce a lighted area of the display whenviewed from the front of the panel (a “normally white” LCD).

To darken a pixel and create an image, a voltage, typically controlledby a thin film transistor, is applied to an electrode in an array oftransparent electrodes deposited on the walls of the cell gap. Theliquid crystal molecules adjacent to the electrode are attracted by thefield produced by the voltage and rotate to align with the field. As themolecules of liquid crystal are rotated by the electric field, thecolumn of crystals is “untwisted,” and the optical axes of the crystalsadjacent to the cell wall are rotated progressively out of alignmentwith the optical axis of the corresponding polarizer progressivelyreducing the local transmittance of the light valve 54 and attenuatingthe luminance of the corresponding pixel. In other words, in a normallywhite twisted nematic device there are generally two modes of operation,one without a voltage applied to the molecules and one with a voltageapplied to the molecules. With a voltage applied (e.g., driven mode) tothe molecules the molecules rotate their polarization axis which resultsin inhibiting the passage of light to the viewer. Similarly, without avoltage applied (e.g., non-driven mode) the polarization axis is notrotated so that the passage of light is not inhibited to the viewer.

Conversely, the polarizers and buffing of the light valve can bearranged to produce a “normally black” LCD having pixels that are dark(light is blocked) when the electrodes are not energized and light whenthe electrodes are energized. Color LCD displays are created by varyingthe intensity of transmitted light for each of a plurality of primarycolor (typically, red, green, and blue) sub-pixels that make up adisplayed pixel.

The aforementioned example was described with respect to a twistednematic device. However, this description is only an example and otherdevices may likewise be used, including but not limited to, multi-domainvertical alignment, patterned vertical alignment, in-plane switching,and super-twisted nematic type LCDs. In addition other devices, such asfor example, plasma displays, organic displays, active matrix organiclight emitting display, electroluminescent displays, liquid crystal onsilicon displays, reflective liquid crystal devices may likewise beused. For such displays the light emitting portion of the display, orportion of the display that permits the display of selected portions oflight may be considered to selectively cause the pixels to providelight.

For an active matrix LCD (AMLCD) the inner surface of the second glasssubstrate 63 is normally coated with a continuous electrode while thefirst glass substrate 61 is patterned into individual pixel electrodes.The continuous electrode may be constructed using a transparentelectrode, such as indium tin oxide. The first glass substrate 61 mayinclude thin film transistors (TFTs) which act as individual switchesfor each pixel electrode (or group of pixel electrodes) corresponding toa pixel (or group of pixels). The TFTs are addressed by a set ofmultiplexed electrodes running along the gaps between the pixelelectrodes. Alternatively the pixel electrodes may be on a differentlayer from the TFTs. A pixel is addressed by applying voltage (orcurrent) to a selected line, which switches the TFT on and allows chargefrom the data line to flow onto the rear pixel electrodes. Thecombination of voltages between the front electrode and the pixelelectrodes sets up a voltage across the pixels and turns the respectivepixels on. The thin-film transistors are typically constructed fromamorphous silicon, while other types of switching devices may likewisebe used, such as for example, metal-insulator-metal diode andpolysilicon thin-film transistors. The TFT array and pixel electrodesmay alternatively be on the top of the liquid crystal material. Also,the continuous electrode may be patterned or portions selectivelyselected, as desired. Also the light sensitive elements may likewise belocated on the top, or otherwise above, of the liquid crystal material,if desired.

Referring to FIG. 2, the active matrix layer may include a set of datalines and a set of select lines. Normally one data line is included foreach column of pixels across the display and one select line is includedfor each row of pixels down the display, thereby creating an array ofconductive lines. To load the data to the respective pixels indicatingwhich pixels should be illuminated, normally in a row-by-row manner, aset of voltages are imposed on the respective data lines 204 whichimposes a voltage on the sources 202 of latching transistors 200. Theselection of a respective select line 210, interconnected to the gates212 of the respective latching transistors 200, permits the voltageimposed on the sources 202 to be passed to the drain 214 of the latchingtransistors 200. The drains 214 of the latching transistors 200 areelectrically connected to respective pixel electrodes and arecapacitively coupled to a respective common line 221 through arespective Cst capacitor 218. In addition, a respective capacitanceexists between the pixel electrodes enclosing the liquid crystalmaterial, noted as capacitances Clc 222 (between the pixel electrodesand the common electrode on the color plate). The common line 221provides a voltage reference. In other words, the voltage data(representative of the image to be displayed) is loaded into the datalines for a row of latching transistors 200 and imposing a voltage onthe select line 210 latches that data into the holding capacitors andhence the pixel electrodes. Alternatively, the display may be operatedbased upon current levels.

Referring to FIG. 3, a schematic layout is shown of the active matrixlayer. The pixel electrodes 230 are generally grouped into a “single”effective pixel so that a corresponding set of pixel electrodes 230 maybe associated with respective color filters (e.g., red, green, blue).The latching transistors 200 interconnect the respective pixelelectrodes 230 with the data lines and the select line. The pixelelectrodes 230 may be interconnected to the common line 221 by thecapacitors Cst 218. The pixels may include any desirable shape, anynumber of sub-pixels, and any set of color filters.

Referring to FIG. 4, the active matrix layer may be constructed using anamorphous silicon thin-film transistor fabrication process. The stepsmay include gate metal deposition (FIG. 4A), a photolithography/etch(FIG. 4B), a gate insulator and amorphous silicon deposition (FIG. 4C),a photolithography/etch (FIG. 4D), a source/drain metal deposition (FIG.4E), a photolithography/etch (FIG. 4F), an ITO deposition (FIG. 4G), anda photolithography/etch (FIG. 4H). Other processes may likewise be used,as desired.

The present inventors considered different potential architectural touchpanel schemes to incorporate additional optical layers between thepolarizer on the front of the liquid crystal display and the front ofthe display. These additional layers include, for example, glass plates,wire grids, transparent electrodes, plastic plates, spacers, and othermaterials. In addition, the present inventors considered the additionallayers with different optical characteristics, such as for example,birefringence, non-birefringence, narrow range of wavelengths, widerange of wavelengths, etc. After an extensive analysis of differentpotential configurations of the touch screen portion added to thedisplay together with materials having different optical properties andfurther being applied to the different types of technologies (e.g.,mechanical switches, crossed beams of infrared light, acoustic surfacewaves, capacitance sensing, and resistive membranes), the presentinventors determined that an optimized touch screen is merely a tradeoffbetween different undesirable properties. Accordingly, the design of anoptimized touch screen is an ultimately unsolvable task. In contrast todesigning an improved touch screen, the present inventors came to therealization that modification of the structure of the active matrixliquid crystal device itself could provide an improved touch screencapability without all of the attendant drawbacks to the touch screenconfiguration located on the front of the display.

Referring to FIG. 5, with particular attention to the latchingtransistors of the pixel electrodes, a black matrix 240 is overlying thelatching transistors so that significant ambient light does not strikethe transistors. Color filters 242 may be located above the pixelelectrodes. Ambient light striking the latching transistors results indraining the charge imposed on the pixel electrodes through thetransistor. The discharge of the charge imposed on the pixel electrodesresults in a decrease in the operational characteristics of the display,frequently to the extent that the display is rendered effectivelyinoperative. With the realization that amorphous silicon transistors aresensitive to light incident thereon, the present inventors determinedthat such transistors within the active matrix layer may be used as abasis upon which to detect the existence of or non-existence of ambientlight incident thereon (e.g., relative values thereto).

Referring to FIG. 6, a modified active matrix layer may include aphoto-sensitive structure or elements. The preferred photo-sensitivestructure includes a photo-sensitive thin film transistor (photo TFT)interconnected to a readout thin film transistor (readout TFT). Acapacitor Cst2 may interconnect the common line to the transistors.Referring to FIG. 7, a black matrix may be in an overlying relationshipto the readout TFT. The black matrix is preferably an opaque material orotherwise the structure of the display selectively inhibiting thetransmission of light to selective portions of the active matrix layer.Preferably the black matrix is completely overlying the amorphoussilicon portion of the readout TFT, and at least partially overlying theamorphous silicon portion of the readout TFT. Preferably the blackmatrix is completely non-overlying the amorphous silicon portion of thephoto TFT, and at least partially non-overlying the amorphous siliconportion of the photo TFT. Overlying does not necessarily denote directcontact between the layers, but is intended to denote in the generalsense the stacked structure of materials. The black matrix is preferablyfabricated on a layer other than the active plate, such as the colorplate. The active plate is normally referred to as the plate supportingthe thin-film transistors. The location of the black matrix on the colorplate (or other non-active plate) results in limited additionalprocessing or otherwise modification the fabrication of the activematrix. In some embodiments, the black matrix inhibits ambient lightfrom impacting the amorphous silicon portion of the readout TFT to anextent greater than inhibiting ambient light from impacting theamorphous silicon portion of the photo TFT. A gate metal, or other lightinhibiting material, may inhibit the photo-sensitive elements from theback light.

Typically the photo-sensitive areas (channels of the transistors) aregenerally rectangular in shape, although other shapes may be used. Theopening in the black matrix is preferably wider (or longer) than thecorresponding channel area. In this manner the channel area and theopening in the black matrix are overlapping, with the opening extendingin a first dimension (e.g., width) greater than the channel area and ina second dimension (e.g., length) less than the channel area. Thisalignment alleviates the need for precise registration of the layerswhile ensuring reasonable optical passage of light to the lightsensitive element. Other relative seizes may likewise be used, asdescribed.

As an example, the common line may be set at a negative voltagepotential, such as −10 volts. During the previous readout cycle, avoltage is imposed on the select line which causes the voltage on thereadout line to be coupled to the drain of the photo TFT and the drainof the readout TFT, which results in a voltage potential across Cst2.The voltage coupled to the drain of the photo TFT and the drain of thereadout TFT is approximately ground (e.g., zero volts) with thenon-inverting input of the operational amplifier connected to ground.The voltage imposed on the select line is removed so that the readoutTFT will turn “off”.

Under normal operational conditions ambient light from the front of thedisplay passes through the black matrix and strikes the amorphoussilicon of the photo TFT. However, if a person touches the front of thedisplay in a region over the opening in the black matrix or otherwiseinhibits the passage of light through the front of the display in aregion over the opening in the black matrix, then the photo TFTtransistor will be in an “off” state. If the photo TFT is “off” then thevoltage across the capacitor Cst2 will not significantly dischargethrough the photo TFT. Accordingly, the charge imposed across Cst2 willbe substantially unchanged. In essence, the voltage imposed across Cst2will remain substantially unchanged if the ambient light is inhibitedfrom striking the photo TFT.

To determine the voltage across the capacitor Cst2, a voltage is imposedon the select line which causes the gate of the readout TFT tointerconnect the imposed voltage on Cst2 to the readout line. If thevoltage imposed on the readout line as a result of activating thereadout TFT is substantially unchanged, then the output of theoperational amplifier will be substantially unchanged (e.g., zero). Inthis manner, the system is able to determine whether the light to thedevice has been inhibited, in which case the system will determine thatthe screen has been touched at the corresponding portion of the displaywith the photo TFT.

During the readout cycle, the voltage imposed on the select line causesthe voltage on the respective drain of the photo TFT and the drain ofthe readout TFT to be coupled to the respective readout line, whichresults in resetting the voltage potential across Cst2. The voltagecoupled to the drain of the photo TFT and the drain of the readout TFTis approximately ground (e.g., zero volts) with the non-inverting inputof the operational amplifier connected to ground. The voltage imposed onthe select line is removed so that the readout TFT will turn “off”. Inthis manner, the act of reading the voltage simultaneously acts to resetthe voltage potential for the next cycle.

Under normal operational conditions ambient light from the front of thedisplay passes through the black matrix and strikes the amorphoussilicon of the photo TFT. If a person does not touch the front of thedisplay in a region over the opening in the black matrix or otherwiseinhibits the passage of light through the front of the display in aregion over the opening in the black matrix, then the photo TFTtransistor will be in an “on” state. If the photo TFT is “on” then thevoltage across the capacitor Cst2 will significantly discharge throughthe photo TFT, which is coupled to the common line. In essence thevoltage imposed across Cst2 will decrease toward the common voltage.Accordingly, the charge imposed across Cst2 will be substantiallychanged in the presence of ambient light. Moreover, there is asubstantial difference in the voltage potential across the holdcapacitor when the light is not inhibited versus when the light isinhibited.

Similarly, to determine the voltage across the capacitor Cst2, a voltageis imposed on the select line which causes the gate of the readout TFTto interconnect the imposed voltage to the readout line. If the voltageimposed on the readout line as a result of activating the readout TFT issubstantially changed or otherwise results in an injection of current,then the output of the operational amplifier will be substantiallynon-zero. The output voltage of the operational amplifier isproportional or otherwise associated with the charge on the capacitorCst2. In this manner, the system is able to determine whether the lightto the device has been uninhibited, in which case the system willdetermine that the screen has not been touched at the correspondingportion of the display with the photo TFT.

Referring to FIG. 8, a layout of the active matrix layer may include thephoto TFT, the capacitor Cst2, the readout TFT in a region between thepixel electrodes. Light sensitive elements are preferably included atselected intervals within the active matrix layer. In this manner, thedevice may include touch panel sensitivity without the need foradditional touch panel layers attached to the front of the display. Inaddition, the additional photo TFT, readout TFT, and capacitor may befabricated together with the remainder of the active matrix layer,without the need for specialized processing. Moreover, the complexity ofthe fabrication process is only slightly increased so that the resultingmanufacturing yield will remain substantially unchanged. It is to beunderstood that other light sensitive elements may likewise be used. Inaddition, it is to be understood that other light sensitive electricalarchitectures may likewise be used.

Referring to FIG. 11, a graph of the photo-currents within amorphoussilicon TFTs is illustrated. Line 300 illustrates a dark ambientenvironment with the gate connected to the source of the photo TFT. Itwill be noted that the leakage currents are low and relatively stableover a range of voltages. Line 302 illustrates a dark ambientenvironment with a floating gate of the photo TFT. It will be noted thatthe leakage currents are generally low and relatively unstable over arange of voltages (significant slope). Line 304 illustrates a lowambient environment with the gate connected to the source of the photoTFT. It will be noted that the leakage currents are three orders ofmagnitude higher than the corresponding dark ambient conditions andrelatively stable over a range of voltages. Line 306 illustrates a lowambient environment with a floating gate of the photo TFT. It will benoted that the leakage currents are generally three orders of magnitudehigher and relatively unstable over a range of voltages (significantslope). Line 308 illustrates a high ambient environment with the gateconnected to the source of the photo TFT. It will be noted that theleakage currents are 4.5 orders of magnitude higher than thecorresponding dark ambient conditions and relatively stable over a rangeof voltages. Line 310 illustrates a high ambient environment with afloating gate of the photo TFT. It will be noted that the leakagecurrents are generally 4.5 orders of magnitude higher and relativelyunstable over a range of voltages (significant slope). With thesignificant difference between the dark state, the low ambient state,and the high ambient state, together with the substantially flatresponses over a voltage range (source-drain voltage), the system mayreadily process the data in a confident manner, especially with the gateconnected to the source. In general, the architecture preferably permitsthe leakage currents to be within one order of magnitude over thecentral 50%, more preferably over the central 75%, of the voltage rangeused for displaying images.

Referring to FIG. 9, under high ambient lighting conditions the photoTFT will tend to completely discharge the Cst2 capacitor to the commonvoltage, perhaps with an offset voltage because of the photo TFT. Inthis manner, all of the photo TFTs across the display will tend todischarge to the same voltage level. Those regions with reduced ambientlighting conditions or otherwise where the user blocks ambient lightfrom reaching the display, the Cst2 capacitor will not fully discharge,as illustrated by the downward spike in the graph. The downward spike inthe graph provides location information related to the region of thedisplay that has been touched.

Referring to FIG. 10, under lower ambient lighting conditions the photoTFT will tend to partially discharge the Cst2 capacitor to the commonvoltage. In this manner, all of the photo TFTs across the display willtend to discharge to some intermediate voltage levels. Those regionswith further reduced ambient lighting conditions or otherwise where theuser blocks ambient light from reaching the display, the Cst2 capacitorwill discharge to a significantly less extent, as illustrated by thedownward spike in the graph. The downward spike in the graph provideslocation information related to the region of the display that has beentouched. As shown in FIGS. 9 and 10, the region or regions where theuser inhibits light from reaching the display may be determined aslocalized minimums. In other embodiments, depending on the circuittopology, the location(s) where the user inhibits light from reachingthe display may be determined as localized maximums or otherwise somemeasure from the additional components.

In the circuit topology illustrated, the value of the capacitor Cst2 maybe selected such that it is suitable for high ambient lightingconditions or low ambient lighting conditions. For low ambient lightingconditions, a smaller capacitance may be selected so that the device ismore sensitive to changes in light. For high ambient lightingconditions, a larger capacitance may be selected so that the device isless sensitive to changes in light. In addition, the dimensions of thephototransistor may be selected to change the photo-leakage current.Also, one set of light sensitive elements (e.g., the photo TFT and thecapacitance) within the display may be optimized for low ambientlighting conditions while another set of light sensitive elements (e.g.,the photo TFT and the capacitance) within the display may be optimizedfor high ambient lighting conditions. Typically, the data from lightsensitive elements for low ambient conditions and the data from lightsensitive elements for high ambient conditions are separately processed,and the suitable set of data is selected. In this manner, the samedisplay device may be used for high and low ambient lighting conditions.In addition, multiple levels of sensitivity may be provided. It is to beunderstood that a single architecture may be provided with a wide rangeof sensitivity from low to high ambient lighting conditions. Inaddition, any suitable alternative architecture may be used for sensingthe decrease and/or increase in ambient light.

Another structure that may be included is selecting the value of thecapacitance so that under normal ambient lighting conditions the chargeon the capacitor only partially discharges. With a structure where thecapacitive charge only partially discharges, the present inventorsdetermined that an optical pointing device, such as a light wand orlaser pointer, might be used to point at the display to furtherdischarge particular regions of the display. In this manner, the regionof the display that the optical pointing device remains pointed at maybe detected as local maximums (or otherwise). In addition, those regionsof the display where light is inhibited will appear as local minimums(or otherwise). This provides the capability of detecting not only theabsence of light (e.g., touching the panel) but likewise those regionsof the display that have increased light incident thereon. Referring toFIG. 12, a graph illustrates local minimums (upward peaks) from addedlight and local maximums (downward peaks) from a lack of light. Inaddition, one set of light sensitive elements (e.g., the photo TFT andthe capacitance) within the display may be optimized for ambientlighting conditions to detect the absence of light while another set oflight sensitive elements (e.g., the photo TFT and the capacitance)within the display may be optimized for ambient lighting conditions todetect the additional light imposed thereon.

A switch associated with the display may be provided to select among aplurality of different sets of light sensitive elements. For example,one of the switches may select between low, medium, and high ambientlighting conditions. For example, another switch may select between atouch sensitive operation (absence of light) and an optical pointingdevice (addition of light). In addition, the optical pointing device maycommunicate to the display, such as through a wire or wirelessconnection, to automatically change to the optical sensing mode. A lightsensor (external photo-sensor to the light sensitive elements in theactive layer) and/or one or more of the light sensitive elements may beused to sense the ambient lighting conditions to select among differentsets of light sensitive elements. Also the sensor and/or one or morelight sensitive elements may be used to select, (1) to sense the absenceof light, (2) select to sense the addition of light, and/or (3) adjustthe sensing levels of the electronics.

In some embodiments the corresponding color filters for (e.g., above)some or all of the light sensitive elements may be omitted or replacedby a clear (or substantially clear) material. In this manner the lightreaching some of the light sensitive elements will not be filtered by acolor filter. This permits those light sensitive elements to sense agreater dynamic range or a different part of the dynamic range thanthose receiving filtered light.

It is noted that the teachings herein are likewise applicable totransmissive active matrix liquid crystal devices, reflective activematrix liquid crystal devices, transflective active matrix liquidcrystal devices, etc. In addition, the light sensitive elements maylikewise be provided within a passive liquid crystal display. Thesensing devices may be, for example, photo resistors and photo diodes.

Alternatively, light sensitive elements may be provided between the rearpolarizing element and the active matrix layer. In this arrangement, thelight sensitive elements are preferably fabricated on the polarizer, orotherwise a film attached to the polarizer. In addition, the lightsensitive elements may be provided on a thin glass plate between thepolarizer and the liquid crystal material. In addition, the black matrixor otherwise light inhibiting material is preferably arranged so as toinhibit ambient light from striking the readout TFT while free frominhibiting light from striking the photo TFT. Moreover, preferably alight blocking material is provided between the photo TFT and/or thereadout TFT and the backlight, such as gate metal, if provided, toinhibit the light from the backlight from reaching the photo TFT and/orthe readout TFT.

Alternatively, light sensitive elements may be provided between thefront polarizing element and the liquid crystal material. In thisarrangement, the light sensitive elements are preferably fabricated onthe polarizer, or otherwise a film attached to the polarizer. Inaddition, the light sensitive elements may be provided on a thin glassplate between the polarizer and the liquid crystal material. The lightsensitive elements may likewise be fabricated within the front electrodelayer by patterning the front electrode layer and including suitablefabrication techniques. In addition, a black matrix or otherwise lightinhibiting material is preferably arranged so as to inhibit ambientlight from striking the readout TFT while free from inhibiting lightfrom striking the photo TFT. Moreover, preferably a light blockingmaterial is provided between the photo TFT and/or the readout TFT andthe backlight, if provided, to inhibit the light from the backlight fromreaching the photo TFT and/or the readout TFT.

Alternatively, light sensitive elements may be provided between thefront of the display and the rear of the display, normally fabricated onone of the layers therein or fabricated on a separate layer providedwithin the stack of layers within the display. In the case of a liquidcrystal device with a backlight the light sensitive elements arepreferably provided between the front of the display and the backlightmaterial. The position of the light sensitive elements are preferablybetween (or at least partially) the pixel electrodes, when viewed from aplan view of the display. This may be particularly useful for reflectivedisplays where the pixel electrodes are opaque. In addition forreflective displays, any reflective conductive electrodes should bearranged so that they do not significantly inhibit light from reachingthe light sensitive elements In this arrangement, the light sensitiveelements are preferably fabricated on one or more of the layers, orotherwise a plate attached to one or more of the layers. In addition, ablack matrix or otherwise light inhibiting material is preferablyarranged so as to inhibit ambient light from striking the readout TFTwhile free from inhibiting light from striking the photo TFT. Moreover,preferably a light blocking material is provided between the photo TFTand/or the readout TFT and the backlight, if provided, to inhibit thelight from the backlight from reaching the photo TFT and/or the readoutTFT.

In many applications it is desirable to modify the intensity of thebacklight for different lighting conditions. For example, in darkambient lighting conditions it may be beneficial to have a dimbacklight. In contrast, in bright ambient lighting conditions it may bebeneficial to have a bright backlight. The integrated light sensitiveelements within the display stack may be used as a measure of theambient lighting conditions to control the intensity of the backlightwithout the need for an additional external photo-sensor. One lightsensitive element may be used, or a plurality of light sensitive elementmay be used together with additional processing, such as averaging.

In one embodiment, the readout line may be included in a periodic mannerwithin the display sufficient to generally identify the location of the“touch”. For example the readout line may be periodically added at each30.sup.th column. Spacing the readout lines at a significant number ofpixels apart result in a display that nearly maintains its previousbrightness because most of the pixel electrodes have an unchanged size.However, after considerable testing it was determined that such periodicspacing results in a noticeable non-uniform gray scale because ofdifferences in the size of the active region of the pixel electrodes.One potential resolution of the non-uniform gray scale is to modify theframe data in a manner consistent with the non-uniformity, such asincreasing the gray level of the pixel electrodes with a reduced size orotherwise reducing the gray levels of the non-reduced size pixelelectrodes, or a combination thereof. While a potential resolution, thisrequires additional data processing which increases the computationalcomplexity of the system.

A more desirable resolution of the non-uniform gray scale is to modifythe display to include a readout line at every third pixel, or otherwisein a manner consistent with the pixel electrode pattern of the display(red pixel, green pixel, blue pixel). Alternatively, a readout line isincluded at least every 12.sup.th pixel (36 pixel electrodes of a red,blue, green arrangement), more preferably at least every 9.sup.th pixel(27 pixel electrodes of a red, blue, green arrangement), even morepreferably at least every 6.sup.th pixel (18 pixel electrodes of a red,blue, green arrangement or 24 pixel electrodes of a red, blue, bluegreen arrangement), and most preferably at least every 3.sup.rd pixel (3pixel electrodes of a red, blue, green arrangement). The readout linesare preferably included for at least a pattern of four times the spacingbetween readout lines (e.g., 12.sup.th pixel times 4 equals 48 pixels,9.sup.th pixel times 4 equals 36 pixels). More preferably the pattern ofreadout lines is included over a majority of the display. The resultingdisplay may include more readout lines than are necessary to accuratelydetermine the location of the “touch”. To reduce the computationalcomplexity of the display, a selection of the readout lines may be freefrom interconnection or otherwise not operationally interconnected withreadout electronics. In addition, to further reduce the computationalcomplexity of the display and to increase the size of the pixelelectrodes, the readout lines not operationally interconnected withreadout electronics may likewise be free from an associated lightsensitive element. In other words, additional non-operational readoutlines may be included within the display to provide a gray scale displaywith increased uniformity. In an alternative embodiment, one or more ofthe non-operational readout lines may be replaced with spaces. In thismanner, the gray scale display may include increased uniformity, albeitwith additional spaces within the pixel electrode matrix.

The present inventors considered the selection of potential pixelelectrodes and came to the realization that the electrode correspondingto “blue” light does not contribute to the overall white transmission tothe extent that the “green” or “red” electrodes. Accordingly, the systemmay be designed in such a manner that the light sensitive elements areassociated with the “blue” electrodes to an extent greater than theirassociation with the “green” or “red” electrodes. In this manner, the“blue” pixel electrodes may be decreased in size to accommodate thelight sensitive elements while the white transmission remainssubstantially unchanged. Experiments have shown that reducing the sizeof the “blue” electrodes to approximately 85% of their original size,with the “green” and “red” electrodes remaining unchanged, results in areduction in the white transmission by only about 3 percent.

While such an additional set of non-operational readout lines providesfor increased uniform gray levels, the reduction of pixel aperturesresults in a reduction of brightness normally by at least 5 percent andpossibly as much as 15 percent depending on the resolution and layoutdesign rules employed. In addition, the manufacturing yield is decreasedbecause the readout line has a tendency to short to its neighboring dataline if the processing characteristics are not accurately controlled.For example, the data line and readout line may be approximately 6-10microns apart along a majority of their length.

Referring to FIG. 13, to increase the potential manufacturing yield andthe brightness of the display, the present inventors came to therealization that the readout of the photo-sensitive circuit and thewriting of data to the pixels may be combined on the same bus line, orotherwise a set of lines that are electrically interconnected to oneanother. To facilitate the use of the same bus line, a switch 418 mayselect between providing new data 420 to the selected pixels and readingdata 414 from the selected pixels. With the switch 418 set tointerconnect the new data 420 with the selected pixels, the data from aframe buffer or otherwise the video data stream may be provided to thepixels associated with one of the select lines. Multiple readoutcircuits may be used, or one or more multiplexed readout circuits maybeused. For example, the new data 420 provided on data line 400 may be 4.5volts which is latched to the pixel electrode 402 and the photo TFT 404by imposing a suitable voltage on the select line 406. In this manner,the data voltage is latched to both the pixel electrode and acorresponding photo-sensitive circuit.

The display is illuminated in a traditional manner and the voltageimposed on the photo TFT 404 may be modified in accordance with thelight incident on the photo-sensitive circuit, as previously described.In the topology illustrated, the photo TFT 404 is normally a N-typetransistor which is reverse biased by setting the voltage on the commonline 408 to a voltage lower than an anticipated voltage on the photo TFT404, such as −10 or −15 volts. The data for the current frame may bestored in a frame buffer for later usage. Prior to writing the data foranother frame, such as the next frame, the data (e.g., voltage) on thereadout TFT 410 is read out. The switch 418 changes between the new data420 to the readout line 414 interconnected to the charge readoutamplifier 412. The select line 406 is again selected to couple theremaining voltage on the photo TFT 404 through the readout TFT 410 tothe data line 400. The coupled voltage (or current) to the data line 400is provided as an input to the charge readout amplifier 412 which iscompared against the corresponding data from the previous frame 422,namely, the voltage originally imposed on the photo TFT 404. Thedifference between the readout line 414 and the data from the previousframe 422 provides an output to the amplifier 412. The output of theamplifier 412 is provided to the processor. The greater the drain of thephoto TFT 404, normally as a result of sensing light, results in agreater output of the amplifier 412. Referring to FIG. 14, an exemplarytiming for the writing and readout on the shared data line 400 isillustrated.

At low ambient lighting conditions and at dark lighting conditions, theintegrated optical touch panel is not expected to operate well to thetouch of the finger because there will be an insufficient (or none)difference between the signals from the surrounding area and the touchedarea. To alleviate the inability to effectively sense at the low anddark ambient lighting conditions a light pen or laser pointer may beused (e.g., light source), as previously described. The light source maybe operably interconnected to the display such as by a wire or wirelesscommunication link. With the light source operably interconnected to thedisplay the intensity of the light source may be controlled, at least inpart, by feedback from the photo-sensitive elements or otherwise thedisplay, as illustrated in FIG. 15. When the display determines thatsufficient ambient light exists, such as ambient light exceeding athreshold value, the light source is turned “off”. In this manner,touching the light source against the display results in the same effectas touching a finger against the display, namely, impeding ambient lightfrom striking the display. When the display determines that insufficientambient light exists, such as ambient light failing to exceed athreshold value, the light source is turned “on”. In this manner,touching or otherwise directing the light from the light source againstthe display results in a localized increase in the received lightrelative to the ambient light level. This permits the display to beoperated in dark ambient lighting conditions or by feedback from thedisplay. In addition, the intensity of the light from the light sourcemay be varied, such as step-wise, linearly, non-linearly, orcontinuously, depending upon the ambient lighting conditions.Alternatively, the light source may include its own ambient lightdetector so that feedback from the display is unnecessary and likewisecommunication between the light source and the display may beunnecessary.

While using light from an external light source while beneficial it maystill be difficult to accurately detect the location of the additionallight because of background noise within the system and variablelighting conditions. The present inventors considered this situation anddetermined that by providing light during different frames, such as oddframes or even frames, or odd fields or even fields, or every thirdframe, or during selected frames, a more defined differential signalbetween the frames indicates the “touch” location. In essence, the lightmay be turned on and off in some manner, such as blinking at a ratesynchronized with the display line scanning or frames. An exemplarytiming for an odd/even frame arrangement is shown in FIG. 16. Inaddition, the illumination of some types of displays involves scanningthe display in a row-by-row manner. In such a case, the differentialsignal may be improved by modifying the timing of the light pulses inaccordance with the timing of the gate pulse (e.g., scanning) for therespective pixel electrodes. For example, in a top-down scanning displaythe light pulse should be earlier when the light source is directedtoward the top of the display as opposed to the bottom of the display.The synchronization may be based upon feedback from the display, ifdesired.

In one embodiment, the light source may blink at a rate synchronizedwith the display line scanning. For example, the light source may usethe same driver source as the image pixel electrodes. In anotherembodiment the use of sequential (or otherwise) frames may be subtractedfrom one another which results in significant different between signaland ambient conditions. Preferably, the light sensitive elements have adynamic range greater than 2 decades, and more preferably a dynamicrange greater than 4 decades. If desired, the system may use twosequential fields of scanning (all lines) subtracted from the next twofields of scanning (all lines) so that all the lines of the display areused.

Another technique for effective operation of the display in dark or lowlevel ambient conditions is using a pen or other device with a lightreflecting surface that is proximate (touching or near touching) thedisplay when interacting with the display. The light from the backlighttransmitted through the panel is then reflected back into thephoto-sensitive element and the readout signal will be greater at thetouch location than the surrounding area.

Referring to FIG. 17, another type of reflective liquid crystal display,typically used on handheld computing devices, involves incorporating alight guide in front of the liquid crystal material, which is normally aglass plate or clear plastic material. Normally, the light guide isconstructed from an opaque material having an index of refractionbetween 1.4 and 1.6, more typically between 1.45 and 1.50, and sometimesof materials having an index of refraction of 1.46. The light guide mayfurther include anti-glare and anti-reflection coatings. The light guideis frequently illuminated with a light source, frequently disposed tothe side of the light guide. The light source may be any suitabledevice, such as for example, a cold cathode fluorescent lamp, anincandescent lamp, and a light emitting diode. To improve the lightcollection a reflector may be included behind the lamp to reflect lightthat is emitted away from the light guide, and to re-direct the lightinto the light guide. The light propagating within the light guidebounces between the two surfaces by total internal reflections. Thetotal internal reflections will occur for angles that are above thecritical angle, measured relative to the normal to the surfaces, asillustrated in FIG. 18. To a first order approximation, the criticalangle .beta. maybe defined by Sin (.beta.)=1/n where n is the index ofrefraction of the light guide. Since the surfaces of the light guide arenot perfectly smooth there will be some dispersion of the light, whichcauses some illumination of the display, as shown in FIG. 19.

The present inventors came to the realization that the critical angleand the disruption of the total internal reflections may be modified insuch a manner as to provide a localized increase in the diffusion oflight. Referring to FIG. 20, one suitable technique for the localizeddiffusion of light involves using a plastic pen to touch the front ofthe display. The internally reflected light coincident with the locationthat the pen touches the display will significantly diffuse and bedirected toward the photo sensitive elements within the display. Theplastic pen, or other object including the finger or the eraser of apencil, preferably has an index of refraction within 0.5, morepreferably within 0.25, of the index of refraction of the light guide.For example, the index of refraction of the light guide may be between1.2 and 1.9, and more preferably between 1.4 and 1.6. With the twoindexes of refraction sufficiently close to one another the disruptionof the internal reflections, and hence amount of light directed towardthe photo-sensitive elements, is increased. In addition, the plastic penpreferably has sufficient reflectivity of light as opposed to beingnon-reflective material, such as for example, black felt.

Referring to FIG. 21, after further consideration the present inventorswere surprised to note that a white eraser a few millimeters away fromthe light guide results in a darkened region with generally consistentoptical properties while a white eraser in contact with the light guideresults in a darkened region with generally consistent opticalproperties together with a smaller illuminated region. In the preferredembodiment, the light sensitive elements are positioned toward the frontof the display in relation to the liquid crystal material (or otherwisethe light valve or electroluminescent material) so that a clearer imagemay be obtained. It is to be understood that any suitable pointingdevice may be used. The illuminated region has an illumination brighterin relation to the remainder of the darkened region. The illuminatedregion may be located by any suitable technique, such as for example, acenter of gravity technique.

In some cases the display screen will have a relatively light coloredregion, such as white or tan, which is used as a virtual button foroperating software. Within this light colored region is typicallytextual information in relatively dark letters. In liquid crystaldisplay technology the light colored region is indicative of lightpassing through the liquid crystal material. Accordingly, if a pointinginstrument includes a generally reflective material the light passingthrough the display may be reflected back through the display. The lightreflected back through the display may be sensed by the light sensitiveelements.

After further consideration of the illuminated region the presentinventors came to the realization that when users use a “touch panel”display, there is a likelihood that the pointing device (or finger) may“hover” at a location above the display. Normally, during this hoveringthe user is not actually selecting any portion of the display, butrather still deciding where to select. In this manner, the illuminatedregion is beneficial because it provides a technique for thedetermination between when the user is simply “hovering” and the userhas actually touched (e.g., “touching”) the display.

In part, the sensitivity to hovering may be related to the lightsensitive elements being primarily sensitive to collimated light whichis inhibited by the finger or other element in proximity to the devicebecause of the alignment of the opening in the black matrix to the pixelelectrodes. To reduce the dependency to collimated light the blackmatrix may include central material aligned with the respective pixelelectrodes so that the light sensitive elements have an increasedsensitivity to non-collimated light (or otherwise non-perpendicular orotherwise angled incident light), as illustrated in FIG. 22. In someembodiments, the openings may be considered a non-continuous opening orotherwise the spatial opening for a particular pixel is noncontinuous.

Another potential technique for the determination between “hovering” and“touching” is to temporally model the “shadow” region (e.g., lightimpeded region of the display). In one embodiment, when the user istypically touching the display then the end of the shadow will typicallyremain stationary for a period of time, which may be used as a basis, atleast in part, of “touching”. In another embodiment, the shadow willtypically enlarge as the pointing device approaches the display andshrinks as the pointing device recedes from the display, where thegeneral time between enlarging and receding may be used as a basis, atleast in part, of “touching”. In another embodiment, the shadow willtypically enlarge as the pointing device approaches the display andmaintain the same general size when the pointing device is touching thedisplay, where the general time where the shadow maintains the same sizemay be used as a basis, at least in part, of “touching”. In anotherembodiment, the shadow will typically darken as the pointing deviceapproaches the display and maintain the same shade when the pointingdevice is touching the display, where the general time where the shadowmaintains the same general shade may be used as a basis, at least inpart, of “touching”.

To further distinguish between the finger or other devices being closeto the display (or touching) or alternatively being spaced sufficientlyapart from the display, a light directing structure may be used. Onesuch light directing structure is shown in FIG. 23. The light directingstructure is preferably included around a portion of the periphery ofthe display and may reflect ambient light across the frontal region ofthe display. The reflected light then reflects off the finger or otherdevice thus increasing the light striking the light sensitive elementwhen the finger or other device is spaced sufficiently apart from thedisplay. The light reflecting off the finger or other device decreaseswhen the finger or other device is near the display because of theangular reflections of light. The differences in the reflected lightstriking the display may be used, at least in part, to detect thetouching of the display or otherwise inhibiting light to the display.

While attempting to consider implementation of such techniques on ahandheld device it came to the inventor's surprise that the displayportion of a handheld device has a refresh rate generally less than therefresh rate of the portion of the handwriting recognition portion ofthe display. The handheld portion of the display may use any recognitiontechnique, such as Palm OS® based devices. The refresh rate of thedisplay is typically generally 60 hertz while the refresh rate of thehandwriting portion of the display is typically generally 100 hertz.Accordingly, the light-sensitive elements should be sampled at asampling rate corresponding with the refresh rate of the respectiveportion of the display.

The technique described with respect to FIG. 20 operates reasonably wellin dark ambient lighting conditions, low ambient lighting conditions,regular ambient lighting conditions, and high ambient lightingconditions. During regular and high ambient lighting conditions, thedisplay is alleviated of a dependency on the ambient lightingconditions. In addition, with such lighting the illumination point ismore pronounced and thus easier to extract. Unfortunately, during thedaytime the ambient light may be sufficiently high causing the detectionof the pointing device difficult. In addition, shades of the ambientlight may also interfere with the detection techniques.

The present inventors considered improving the robustness of thedetection techniques but came to the realization that with sufficient“noise” in the system the creation of such sufficiently robusttechniques would be difficult. As opposed to the traditional approach ofimproving the detection techniques, the present inventors came to therealization that by providing light to the light guide of a limitedselection of wavelengths and selectively filtering the wavelengths oflight within the display the difference between touched and un-touchedmay be increased. As an initial matter the light from the light sourceprovided to the light guide is modified, or otherwise filtered, toprovide a single color. Alternatively, the light source may providelight of a range of wavelengths, such as 600-700 nm, or 400-500 and530-580, or 630. Typically, the light provided to the light guide has arange of wavelengths (in any significant amount) less than white lightor otherwise the range of wavelengths of ambient light. Accordingly,with the light provided to the light guide having a limited color gamut(or reduced color spectrum) the touching of the pointing device on thedisplay results in the limited color gamut light being locally directedtoward the light-sensitive elements. With a limited color gamut lightbeing directed toward the display as a result of touching the lightguide (or otherwise touching the front of the display), a color filtermay be included between the light guide and the light-sensitive elementsto filter out at least a portion of the light not included within thelimited color gamut. In other words, the color filter reduces thetransmission of ambient light to an extent greater than the transmissionof light from the light source or otherwise within the light guide. Forexample, the ambient light may be considered as “white” light while thelight guide has primarily “red” light therein. A typical transmission ofa red color filter for ambient white light may be around 20%, while thesame color filter will transmit about 85% of the red light. Preferablythe transmission of ambient light through the color filter is less than75% (greater than 25% attenuation) (or 60%, 50%, 40%, 30%) while thetransmission of the respective light within the light guide is greaterthan 25% (less than 25% attenuation) (or 40%, 50%, 60%, 70%), so that inthis manner there is sufficient attenuation of selected wavelengths ofthe ambient light with respect to the wavelengths of light within thelight guide to increase the ability to accurately detect the touching.

In another embodiment, the light source to the light guide may include aswitch or otherwise automatic modification to “white” light whenoperated in low ambient lighting conditions. In this manner, the displaymay be more effective viewed at low ambient lighting conditions.

In another embodiment, the present inventors determined that if thelight source providing light to the display was modulated in somefashion an improvement in signal detection may be achieved. For example,a pointing device with a light source associated therewith may modulatethe light source in accordance with the frame rate of the display. Witha frame rate of 60 hertz the pointing device may for example modulatethe light source at a rate of 30 hertz, 20 hertz, 10 hertz, etc. whichresults in additional light periodically being sensed by the lightsensitive elements. Preferably, the light source is modulated(“blinked”) at a rate synchronized with the display line scanning, anduses the same raw drivers as the image thin-film transistors. Theresulting data may be processed in a variety of different ways.

In one embodiment, the signals from the light sensitive elements areused, as captured. The resulting improvement in signal to backgroundratio is related to the pulse length of the light relative to the frametime. This provides some additional improvement in signal detectionbetween the light generated by the pointing device relative to theambient light (which is constant in time).

In another embodiment, multiple frames are compared against one anotherto detect the presence and absence of the additional light resultingfrom the modulation. In the case of subsequent frames (sequential ornon-sequential), one without additional light and one with additionallight, the data from the light sensitive elements may be subtracted fromone another. The improvement in signal to background ratio is related tothe periodic absence of the additional light. In addition, thisprocessing technique is especially suitable for low ambient lighting andhigh ambient lighting conditions. Preferably the dynamic range of thesensors is at least 4 decades, and two sequential frames with additionallight and two sequential frames without additional light are used sothat all of the scanning lines are encompassed. When the system chargesa sensor it takes a whole frame for it to discharge by the light. Sincethe first line will start at time zero and take a frame time, the lastline will be charged after almost a frame and will take another frametime to discharge. Therefore, the system should preferably use twoframes with additional illumination and then two frames withoutadditional illumination.

While the light sensitive elements may be used to determine “touch” andthe location of the “touch”, it is sometimes problematic to distinguishbetween “hovering” and “touch”. To assist in the determination of actualtouching of the display a pressure based mechanism may be used. Onepressure based mechanism may include pressure sensitive tape between apair of layers of the display or between the display and a support forthe display. Another pressure based mechanism may include an electricalor magnetic sensor operably connected to the display. In either case,the pressure based mechanism provides a signal to the displayelectronics indicating the sensing of pressure (e.g., touch) oralternatively the absence of pressure (e.g., non-touch).

Referring to FIG. 24, one configuration of an elongate light emittingdevice includes an infra-red light emitting diode that periodically orcontinuously emits an infra-red beam. The infra-red beam is transmittedfrom the light emitting device and reflects off the display. When thelight pen is spaced sufficiently far from the display the reflectedinfra-red beam will not strike the light pen. When the light pen isspaced sufficiently close to the display the reflected infra-red beamwill strike the light pen and is sensed by an infra-red sensor withinthe light pen. Infra-red light is preferred, while any suitablewavelength may be used that the light sensitive elements of the displayare generally insensitive to. When the infra-red sensor senses thereflected infra-red light the visible light emitting diode is turned onto illuminate the pixel. The visible light emitting diode preferablyprovides a wavelength that the light sensitive elements of the displayare sensitive to. After a predetermined duration or otherwise while theinfra-red light is not being sensed by the infra-red sensor the visiblelight emitting diode is turned off. In this manner, battery power withinthe light pen is conserved. In addition, the edge or shape of thevisible light from the visible light emitting diode may be used todetermine the spacing between the light pen and the display. Also, thebeam from the visible light emitting diode may be varied based upon thesignal sensed by the infra-red sensor.

Upon reconsidering the display with a light guide, as illustrated inFIG. 17, the present inventors realized that those portions of the lightguide that are in contact with the high portion of the user'sfingerprints will tend to diffuse and scatter light toward the lightsensitive elements, as illustrated in FIG. 25. Those portions of thelight guide that are not in contact with the high portion of the user'sfingerprints (i.e., valleys) will not tend to diffuse and scatter lighttoward the light sensitive elements. Depending on the thickness of thelight guide and liquid crystal material together with the density of thelight sensitive elements, the details observable in the fingerprint willvary. To increase the ability to detect the fingerprint, the display maybe designed with multiple densities of light sensitive elements. Forexample, in the region where the fingerprint is normally located thedensity of the light sensitive elements may be increased such asincluding a light sensitive element at every sub-pixel. In this manner,the display includes multiple densities of light sensitive elements.Moreover, the display may likewise be used for sensing other items, suchas for example, bar codes.

In some cases there may be excessive parallax which causes a smearedimage to be detected. In addition, oil and sweat on the fingers may tendto reduce the contrast between the high and low point of the user'sfingerprint. Referring to FIG. 26, a separate sensor structure may beincluded within the display. The sensor structure may include a lensbetween the light guide and the light sensitive elements. The lens maybe any suitable lens structure, such as for example, a small focus lensor a SELFOC lens (variable index of refraction fiber optics). The colorfilters in the fingerprint sensing region may be omitted, f desired.

To increase the ability of the light sensitive elements to detect thedetails that may be present within a fingerprint it is desirable to havea greater density of light sensitive elements in the region where thefingerprint is to be sensed. In this manner the display may include two(or more) different densities of light sensitive elements across thedisplay. Alternatively, the light guide and lens may be omitted and ahigh density of light sensitive elements included for fingerprintsensing. It is to be understood that other items may likewise be sensedwith the high density light sensitive elements.

The light sensitive elements will tend to observe very high ambientlighting conditions when the finger is not present rendering theirability to detect high contrast images difficult. To increase theeffective sensitivity of the light sensitive elements a color filter maybe provided in an overlying relationship to the light sensitiveelements. In addition, the light source may be selected in relation tothe color filter. For example, a blue filter may be used together with ablue light source. In addition, the illumination may be modulated andsynchronized with the sensors. In this case, the light source may beilluminated with relatively short pulses together with the triggering ofthe sensing by the light sensitive elements. In the case where the lightis pulsed in relation to the frame rate it is preferably pulsed at halfthe frame rate. In this manner the light pulses will be ensured to besensed during different frames. Also, a light inhibiting material mayclosely surround the region where the finger is locate to reduce strayambient light, thus increasing contrast.

For the detection of fingerprints, sweat and oil from the fingers mayimpact the ability to accurate sense the fingerprint. Sweat is primarilywater with an index of refraction of approximately 1.3. Typical glasshas an index of refraction of approximately 1.5 which is sufficientlydifferent than 1.3, and accordingly sweat will not significantlynegatively impact image sensing. However, oils have an index ofrefraction of approximately 1.44 to 1.47 which is considerably closer to1.5, and accordingly oil will tend to significantly impact imagesensing. In order to improve the contrast, hence image sensing, theglass may be replaced with glass having a higher index of refraction orglass coated with a material having a higher index of refraction, suchas for example, 1.55 or more.

Attempting to record signatures may be problematic with many touchsensitive displays because the response time of the recording system isinadequate or otherwise the software has a slow sampling rate. However,the user normally prefers immediate feedback. Referring to FIG. 27 thedisplay may include a signature portion that includes a memorymaintaining material. The memory maintaining material may sense thewriting of the signature, such as by pressure exerted thereon or lightsensitive material. For example, the memory material may be a pair offlexible layers with fluid there between that is displaced while writingthe signature. After writing the name the user may press a button orotherwise touch another portion of the display indicating that thesignature is completed. Alternatively, the display may sense thesignature by a sufficient change in the optical properties of the memorymaintaining material. The display may then capture an image of thesignature. For example, the image of the signature may come from ambientlight passing through the memory maintaining material or otherwise fromlight reflected off the memory maintaining material, especially when thedisplay is in transmissive (“white”) mode. The signature may be clearedin any manner, such as electrical erasure or physical erasure by anysuitable mechanism. In this manner, the temporal limitations of writinga signature are reduced.

Alternatively the display may include a signature mode that captures theuser's signature over several frames. The signature may be captured on apredetermined region of the display r otherwise any portion of thedisplay. The system detects the decrease in ambient light over a seriesof frames as the signature is written. In this manner the “path” of thesignature maybe determined and thereafter used in any suitable manner.

In many cases the user desires the tactile response of pen pressureagainst the display. In this manner, the user has greater comfort withthe pen and display for writing, drawing, or otherwise indicating aresponse. Referring to FIG. 28, the pen may include a movable opticalpath (e.g., a fiber optic bundle) with respect to the pen that extendsand retracts based upon pressure exerted on the display. A lightemitting diode provides a beam of light that is channeled through theoptical path. As it may be observed, when the optical path is in aretracted state that more light passes through the optical path thanwhen the optical path is in an extended state. In this manner, the lightsensitive elements may detect the intensity of the transmitted light anddetermine the pressure that is being exerted against the display by theuser. This capability is particular useful for pressure sensitiveapplications, such as Photoshop by Adobe.

Referring to FIG. 29 another pen pressure embodiment is illustrated. Acylindrical tubular tip portion is movable with respect to the pen thatextends and retracts based upon pressed exerted on the display. Withinthe cylindrical tubular tip portion is a lens. The lens focuses thelight emitted from a light emitting diode, which preferably ismaintained stationary with respect to the pen and/or moves with respectto the cylindrical tubular tip portion. Alternatively, the lightemitting diode may be connected to the tip portion of the pen. As it maybe observed, when the cylindrical tubular tip portion is in a retractedstate light may be more focused on the display (light sensitiveelements) than when the cylindrical tubular tip portion is in anextended state. The lens may be modified so that it operates in areversed manner. The focus of the beam may be detected in any suitablemanner by the light sensitive elements (e.g., size and/or intensity) todetermine the pressure that is being exerted against the display by theuser.

Referring to FIG. 30 another pen pressure embodiment is illustrated. Acylindrical tubular tip portion or optical light guide is movable withrespect to the pen that extends and retracts based upon pressure exertedon the display. As the cylindrical tubular tip portion moves withrespect to the pen the resistance of a variable resistive elementchanges. The variable resistive element is interconnected to a lightemitting diode which changes intensity based upon the change in thevariable resistance. The intensity of light sensed by the lightsensitive elements, or otherwise the change in intensity sensed by thelight sensitive elements, may be used to determine the pressure that isbeing exerted against the display by the user.

Referring to FIG. 31 another pen pressure embodiment is illustrated. Anoptional lens focuses a beam from a light emitting diode. When the penis farther from the display a larger spot size and/or intensity issensed by the light sensitive elements with respect to when the pen iscloser to the display. The intensity of the light and/or the size of thespot sensed by the light sensitive elements or otherwise the change inintensity and/or size may be used to determine the pressure that isbeing exerted against the display by the user.

Referring to FIG. 32, a modified light guide type structure is shown. Apolarizer may be included on the lower surface of the light guide. Inaddition, other polarizers may be included on the front and rearsurfaces of the liquid crystal display. The polarizer on the lowersurface of the light guide preferably matches (.+−0.5 degrees, .+−0.2degrees) the orientation of the polarizer on the front surface of theliquid crystal display. The polarizers may include anti-reflectioncoatings if desired. With the top two polarizers being included togetherwith proper alignment between them more of the light from the pen willtend to pass through to the liquid crystal display.

Referring to FIG. 33 an exemplary image processing technique isillustrated for processing the data from the display to determine thelocation of the touch. The display is initially calibrated by acalibration image module. For calibration, an image is obtained with thedisplay covered by a black cloth or otherwise blocked from receivingambient light. The black reference image may be referred to as I0. Thenan image may be obtained under normal uniform (e.g., without significantshadows or light spots) ambient lighting conditions and referred to asI1. A comparison between I0 and I1 may be performed to calibrate thedisplay.

During operation, the image is initially captured by a capture imagemodule. For example, a 60.times.60 sensor matrix may be captured.Optionally a set of consecutive frames may then be averaged by anaverage frame module in order to reduce the noise in the signal, such asfor example four frames.

An AGC module may perform an automatic gain control function in order toadjust for an offset in the value. One particular implementation may bedescribed by the following equation: Ie={(Is−I0)/(I1−I0)}. Ie is thesignal after equalization. “Is” is the sensor signal as captured orafter averaging. I0 is the stored sensor reading for the black referenceimage. I1 is the stored sensor reading for the bright (e.g., ambientlighting conditions) reference image. The equalization module uses I0 toadjust for the potential non-zero value at dark conditions. Inparticular, this adjustment is made by the calculation of Is−I0. Theresulting comparison (e.g., division) of the captured signal versus thestored bright reference signals adjusts the level of the signal.Moreover, the output of the equalization is a normalized signal with arange from 0 to 1 in the case that Is<I1. Ie may then be used to adjustthe gain of the output of the average frame module to captured imagevalue. The AGC module thus effectively corrects for dark levelnon-uniformity and for sensor gain non-uniformity.

A smoothing module may be used to average proximate values together tocompensate for non-uniformity in the characteristics of the display. Asuitable filter is an averaging of the 8 adjacent pixels to a centralpixel. Inherently the system provides relatively sharp edges from thesignals which may be used directly. However, with the capability in someembodiments of detecting positive and negative outputs it was determinedthat using an edge detection module is beneficial. A preferred edgedetection technique uses a 3.times.3 matrix, such as for example:{(−1−1−1) (−1−8−1) (−1−1−1)}. The effect of the edge detection module isto enhance or otherwise determine those regions of the image thatindicate an edge. Other edge detection techniques may likewise be used,such as for example, a Sobel edge detection technique, a 1.sup.stderivative technique, and a Robert's cross technique.

A threshold module may be used to set all values below a predeterminedthreshold value to zero, or otherwise indicate that they are not edgesor regions being touched. In the case that the system provides negativevalues the predetermined threshold value may be less than an absolutevalue. The use of threshold values assists in the discrimination betweenthe end of the finger touching the display and the shade from the handitself. If there are an insufficient number of pixels as a result of thethreshold module that are non-thresholded then the system returns to thestart.

If there are a sufficient number of pixels as a result of the thresholdmodule that are non-thresholded then the system determines the largestregion of non-thresholded values using a max location module. In thismanner, smaller regions of a few values may be removed so that thepredominant region of non-thresholded values may be determined. A centerof gravity module may be used to determine the center of the maximumregion from the max location module. An x-y coordinates module may beused to provide the x and y display coordinates and a plot cross modulemaybe used to display a cross on the display at the x-y coordinates ofthe center of gravity. The cross module may provide data regarding theexistence of the “touch” and its location to the system and returncontrol back to the start.

Periodically the display may become scratched or otherwise a foreignobject will be stuck to the front of the display. In this case thedisplay will tend to provide false readings that the scratch or foreignobject is indicative of a touch. In order to reduce the effects ofscratches and foreign objects one or more bright reference images I1 maybe obtained over a period of time. The set of one or more brightreference images I1 may be averaged if an insubstantial differenceexists between the images. This reduces the likelihood that the displaywas touched during one or more of the image acquisitions. In the eventthat the images are substantially different then the images may bereacquired until an insubstantial difference exists.

When operated at low ambient lighting conditions it is difficult todetect the touching of the display. While under normal ambient lightingconditions it is desirable to adjust the output of the display inrelation to the black reference image (I0). However, it has beendetermined that under low ambient lighting conditions it is desirable toadjust the output of the display in relation to the bright referenceimage (I1). Accordingly, one particular implementation may be describedby the following equation: Ie={(Is−I1)/(I1−I0)}. The switching to nightmode (low ambient lighting conditions) may be automatic or in responseto a switch.

All references cited herein are hereby incorporated by reference.

The terms and expressions which have been employed in the foregoingspecification are used therein as terms of description and not oflimitation, and there is no intention, in the use of such terms andexpressions, of excluding equivalents of the features shown anddescribed or portions thereof, it being recognized that the scope of theinvention is defined and limited only by the claims which follow.

We claim:
 1. A display device including a surface for viewing an image,the display device comprising: a stack of layers including: a pluralityof display thin film transistors (TFTs) located in a viewable area ofthe display device, configured to display the image, each of theplurality of display TFTs including one display TFT channel of aplurality of display TFT channels, a plurality of display pixels locatedin the viewable area of the display device and including the pluralityof display TFTs; a plurality of photo-sensitive TFTs located in theviewable area of the display device and including: at least a firstselectable set of the plurality of photo-sensitive TFTs, and at least asecond selectable set of the plurality of photo-sensitive TFTs, each ofthe plurality of photo-sensitive TFTs including the display TFT channelof one of the plurality of display TFTs, each of the plurality ofphoto-sensitive TFTs configured to detect a touch object contacting orhovering over the surface, the first selectable set of the plurality ofphoto-sensitive TFTs having at least a first level of light sensitivityto detect a reduction in ambient light, the second selectable set of theplurality of photo-sensitive TFTs having at least a second level oflight sensitivity to detect an increase in the ambient light, the firstlevel of light sensitivity and the second level of light sensitivitybeing non-identical; a plurality of readout TFTs coupled to theplurality of photo-sensitive TFTs and configured to output a pluralityof signals representative of the detected touch object; and logicconfigured to receive the plurality of signals and determine a locationof the detected touch object based on the plurality of signals.
 2. Thedisplay device as recited in claim 1, further comprising: a black matrixlayer including non-continuous openings overlaying portions of theplurality of display TFT channels, wherein centers of the non-continuousopenings are offset from centers of the plurality of display TFTchannels.
 3. The display device as recited in claim 1, wherein the logicis further configured to: select between the first selectable set of theplurality of photo-sensitive TFTs and the second selectable set of theplurality of photo-sensitive TFTs based on a current lighting condition.4. The display device as recited in claim 3, wherein the currentlighting condition includes a first lighting condition and a secondlighting condition, wherein the first selectable set of the plurality ofphoto-sensitive TFTs is configured to detect the touch object in thefirst lighting condition, wherein the second selectable set of theplurality of photo-sensitive TFTs is configured to detect the touchobject in the second lighting condition, wherein the first level oflight sensitivity is less than the second level of light sensitivity,and wherein the first lighting condition includes higher intensity lightthan the second lighting condition.
 5. The display device as recited inclaim 1, wherein each of the photo-sensitive TFTs included in the firstselectable set of the plurality of photo-sensitive TFTs is coupled to afirst capacitor having a first capacitance, wherein each of thephoto-sensitive TFTs included in the second selectable set of theplurality of photo-sensitive TFTs is coupled to a second capacitorhaving a second capacitance, wherein the first level of lightsensitivity is higher than the second level of light sensitivity, andthe first capacitance is less than the second capacitance.
 6. Thedisplay device as recited in claim 3, wherein the logic is furtherconfigured to automatically select between the first selectable set ofthe plurality of photo-sensitive TFTs and the second selectable set ofthe plurality of photo-sensitive TFTs.
 7. The display device as recitedin claim 1, further comprising: a switch configured to select betweenthe first selectable set of the plurality of photo-sensitive TFTs andthe second selectable set of the plurality of photo-sensitive TFTs basedon a detected lighting condition; and a light sensor, separate anddistinct from the plurality of photo-sensitive TFTs, configured todetect the lighting condition.
 8. The display device as recited in claim1, further comprising: a switch configured to select between the firstselectable set of the plurality of photo-sensitive TFTs and the secondselectable set of the plurality of photo-sensitive TFTs based on one ofa plurality of operating modes, the plurality of operating modesincluding: sensing an absence of light, sensing an addition of light,and adjusting sensing levels of circuitry.
 9. The display device asrecited in claim 1, further comprising: a plurality of capacitors, eachof the plurality of capacitors coupled to one of the plurality ofphoto-sensitive TFTs; and sense circuitry configured to detect anabsence of a change in charge across each of the plurality of capacitorswhen the touch object is contacting or hovering above the surface. 10.The display device as recited in claim 1, further comprising: a blackmatrix layer including: a plurality of openings configured to allowlight to pass through, at least a first portion of the plurality ofopenings at least partially overlays the plurality of photo-sensitiveTFTs, and at least a second portion of the plurality of openingsoverlays at least partially the plurality of display pixels; and aplurality of regions of a material configured to prevent light frompassing through, at least a first portion of the plurality of regions ofthe material overlays the plurality of readout TFTs, and at least asecond portion of the plurality of regions of the material overlays theplurality of display TFTs.
 11. The display device as recited in claim 1,wherein the plurality of photo-sensitive TFTs is configured to detect alighting condition, the device further comprising: a switch configuredto select between the first selectable set of the plurality ofphoto-sensitive TFTs and the second selectable set of the plurality ofphoto-sensitive TFTs based on the detected lighting condition.
 12. Thedisplay device as recited in claim 1, further comprising: a color filterlayer included in the stack of layers between the surface and at least aportion of the first selectable set of the plurality of photo-sensitiveTFTs, a portion of the second selectable set of the plurality ofphoto-sensitive TFTs, or both.
 13. A method for operating a displaydevice, the method comprising: displaying an image from a plurality ofdisplay pixels included in a stack of layers, the plurality of displaypixels located in a viewable area of the display device and including aplurality of display thin film transistors (TFTs); detecting a touchobject contacting or hovering over a surface of the display device by:configuring a plurality of photo-sensitive TFTs for detecting light, theplurality of photo-sensitive TFTs associated with the stack of layersand located in the viewable area of the display device, selecting one ormore of a first selectable set of the plurality of photo-sensitive TFTsand a second selectable set of the plurality of photo-sensitive TFTs,the first selectable set of the plurality of photo-sensitive TFTs havinga first level of light sensitivity to detect a reduction in ambientlight, the second selectable set of the plurality of photo-sensitiveTFTs having a second level of light sensitivity to detect an increase inthe ambient light, the first level of light sensitivity and the secondlevel of light sensitivity being non-identical, and detecting an absenceof a change in charge across a plurality of capacitors, the plurality ofcapacitors coupled to the plurality of photo-sensitive TFTs, when thetouch object is contacting or hovering over the surface, wherein thechange in the charge across the plurality of capacitors is associatedwith the detected light; and determining a location of the touch objectcontacting or hovering over the surface based at least on the detectedabsence of the change in charge across the plurality of capacitors. 14.The method as recited in claim 13, further comprising: separatelyprocessing data from the first selectable set of the plurality ofphoto-sensitive TFTs and data from the second selectable set of theplurality of photo-sensitive TFTs.
 15. The method as recited in claim13, further comprising: configuring the first selectable set of theplurality of photo-sensitive TFTs for detecting an absence of the light;and configuring the second selectable set of the plurality ofphoto-sensitive TFTs for detecting an additional light.
 16. The methodas recited in claim 13, further comprising: detecting a lightingcondition, wherein the selection of one or more of the first selectableset of the plurality of photo-sensitive TFTs and the second selectableset of the plurality of photo-sensitive TFTs is based on the detectedlighting condition.
 17. The method as recited in claim 13, wherein theselection of one or more of the first selectable set of the plurality ofphoto-sensitive TFTs and the second selectable set of the plurality ofphoto-sensitive TFTs is automatic.
 18. The method as recited in claim13, further comprising: configuring the first selectable set of theplurality of photo-sensitive TFTs for a first level of ambient lightingconditions; and configuring the second selectable set of the pluralityof photo-sensitive TFTs for a second level of ambient lightingconditions, the second level greater than the first level.
 19. Themethod as recited in claim 13, further comprising: detecting a lightingcondition by determining an amount of discharge of at least theplurality of capacitors; and classifying the detected lighting conditionbased on the determined amount of discharge.
 20. The method as recitedin claim 13, wherein the selection of one or more of the firstselectable set of the plurality of photo-sensitive TFTs and the secondselectable set of the plurality of photo-sensitive TFTs is based on oneof a plurality of operating modes, the plurality of operating modesincluding: detecting an absence of the light, sensing an addition oflight, and adjusting sensing levels of circuitry.
 21. The display deviceas recited in claim 9, wherein the sense circuitry is further configuredto detect a presence of the change in the charge across each of theplurality of capacitors in an absence of the touch object contacting orhovering above the surface.
 22. The display device as recited in claim1, wherein each of the plurality of photo-sensitive TFTs and each of theplurality of readout TFTs are turned on and off by a controlled source.23. The display device as recited in claim 1, wherein the plurality ofphoto-sensitive TFTs are located on a separate layer in the stack oflayers than display sub-pixels of the plurality of display pixels. 24.The device as recited in claim 1, wherein the plurality of display TFTsassociated with the first selectable set of the plurality ofphoto-sensitive TFTs are photo TFTs having a first dimension, whereinthe plurality of display TFTs associated with the second selectable setof the plurality of photo-sensitive TFTs are photo TFTs having a seconddimension, wherein the first dimension and the second dimension arenon-identical.